Update to LMEval allowOnline setting instructions #901
Conversation
WalkthroughAdds a new assembly documenting external resource access for LMEval jobs, includes two new procedure modules (CLI and web console), updates the main evaluation assembly to include the new section, and revises LM-Eval global settings in setup docs (renamed keys, defaults changed, procedural steps removed). Changes
Sequence Diagram(s)sequenceDiagram
participant Admin
participant OpenShift/Cluster
participant TrustyAI Operator
participant LMEvalJob
Admin->>OpenShift/Cluster: Patch trustyai-service-operator-config (lmes-allow-online/code-execution=true)
Admin->>OpenShift/Cluster: Restart operator deployment
OpenShift/Cluster->>TrustyAI Operator: Apply new config
Admin->>OpenShift/Cluster: Create/Patch LMEvalJob (allowOnline/allowCodeExecution=true)
TrustyAI Operator->>LMEvalJob: Reconcile with updated settings
LMEvalJob->>External Resources: Access datasets/tokenizers (if enabled)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (2)
modules/setting-up-lmeval.adoc (1)
75-83: Ensure ConfigMap key islmes-allow-code-execution, notlmes-code-executionThe console and CLI procedures use
lmes-allow-code-execution, so we should update all docs for consistency:• modules/setting-up-lmeval.adoc (around line 79): change
lmes-code-execution→lmes-allow-code-execution
• modules/using-llm-as-a-judge-metrics-with-lmeval.adoc (around line 68): changelmes-code-execution→lmes-allow-code-executionReplace the table rows in each file as follows:
-| `lmes-code-execution` -| true -| Determines whether LMEval jobs can set the `trust remote code` mode to `on`. +| `lmes-allow-code-execution` +| true +| Determines whether LMEval jobs can set the `trust remote code` mode to `on`.modules/updating-lmeval-job-configuration-using-the-cli.adoc (1)
77-83: Missing step: set per-joballowOnline/allowCodeExecutionon the LMEvalJobCluster-level switches alone are not sufficient. Users must also set the flags on their LMEvalJob resource.
oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ---- + +. Enable online access and/or code execution on the specific `LMEvalJob`: ++ +[source,sh] +---- +# Patch an existing job (example name: example-lmeval) +oc patch lmevaljob example-lmeval -n <user-or-target-namespace> \ + --type merge -p '{"spec":{"allowOnline":true,"allowCodeExecution":true}}' +---- ++ +[source,yaml] +---- +# Or define it in YAML when creating the job +apiVersion: trustyai.opendatahub.io/v1alpha1 +kind: LMEvalJob +metadata: + name: example-lmeval + namespace: <user-or-target-namespace> +spec: + allowOnline: true + allowCodeExecution: true +----
🧹 Nitpick comments (4)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc (1)
8-11: Tighten wording, link style, and clarify independent togglesMinor grammar and clarity improvements; also make it clear that online access and code execution can be enabled independently.
- LMEval Jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, `LMEvalJob` can be configured to automatically download them. + LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it might require access to external resources, for example, task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face^]. If you trust the source and have reviewed the content of these artifacts, LMEval jobs can be configured to automatically download them. -Follow the steps below to enable online access and remote code execution for LMEval Jobs. Choose to update these settings by using either the CLI or in the console. +Follow the steps below to enable online access and/or remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or the console. Enable only what you require for your use case.modules/updating-lmeval-job-configuration-using-the-console.adoc (1)
53-66: Tighten step wording; remove duplicationThe sentence duplicates “in the LMEvalJob custom resource”.
-. Add the following fields to the `LMEvalJob` custom resource to enable online access in the LMEvalJob custom resource: +. Add the following fields to the `LMEvalJob` to enable online access and code execution for this job:modules/updating-lmeval-job-configuration-using-the-cli.adoc (2)
9-9: Minor wording“In the CLI” → “using the CLI”.
-Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes in the CLI for LMEval Jobs. +Follow these steps to enable online access (`allowOnline`) and remote code execution (`allowCodeExecution`) modes using the CLI for LMEval Jobs.
57-59: Clarify you can enable either capability independentlyReaders may think both must be set to true together. Suggest clarifying “and/or” and showing how to set only one.
-. Update the settings for the TrustyAI operator to enable external connectivity and remote code execution: +. Update the settings for the TrustyAI operator to enable external connectivity and/or remote code execution: @@ -oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ ---type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' +oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ +--type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' +# Example: enable online only +# oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ +# --type merge -p '{"data":{"lmes-allow-online":"true"}}'
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
.DS_Storeis excluded by!**/.DS_Store
📒 Files selected for processing (5)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc(1 hunks)assemblies/evaluating-large-language-models.adoc(1 hunks)modules/setting-up-lmeval.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-cli.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-console.adoc(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: bredamc
PR: opendatahub-io/opendatahub-documentation#860
File: modules/using-llm-as-a-judge-metrics-with-lmeval.adoc:62-65
Timestamp: 2025-07-16T09:05:50.422Z
Learning: In the opendatahub-documentation project, bredamc prefers comprehensive error detection in code reviews. Continue to point out potential errors whether they are related to the code changes in the PR or not, but clearly distinguish between errors introduced in the PR versus errors in existing code.
🔇 Additional comments (3)
assemblies/evaluating-large-language-models.adoc (2)
20-21: Placement of new section looks correctIncluding the new section between setup and job execution reads well and keeps the flow logical.
20-21: Confirm assembly-in-assembly include is supported by your build pipelineYou’re including an ASSEMBLY file inside another ASSEMBLY. Some pipelines expect assemblies to include modules only. Please confirm this builds cleanly; otherwise, consider inlining the two procedure modules here or converting the included file to a CONCEPT/REFERENCE module.
modules/updating-lmeval-job-configuration-using-the-cli.adoc (1)
57-83: Add a post-patch verification snippet to confirm settingsTo ensure your ConfigMap update and rollout took effect, append the following under the restart step in
modules/updating-lmeval-job-configuration-using-the-cli.adoc:[source,sh]
Verify ConfigMap values in your cluster
oc get cm trustyai-service-operator-config -n redhat-ods-applications
-o jsonpath='{.data.lmes-allow-online}{"\n"}{.data.lmes-allow-code-execution}{"\n"}'Verify a sample LMEvalJob has the flags set
oc get lmevaljob -n
-o jsonpath='{.spec.allowOnline}{"\n"}{.spec.allowCodeExecution}{"\n"}'Replace
<example-lmeval-name>and<namespace>with your actual job name and namespace, then run these commands against your cluster to confirm bothlmes-allow-onlineandlmes-allow-code-executionare enabled.
modules/updating-lmeval-job-configuration-using-the-console.adoc
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
modules/updating-lmeval-job-configuration-using-the-cli.adoc (4)
11-14: Enhance the security advisory with concrete risks and guidanceConsider briefly calling out typical risks (untrusted code execution, data exfiltration via outbound egress) and recommending mitigations (restrict egress via NetworkPolicy/EgressFirewall, use allowlists).
Apply this minimal tweak:
[IMPORTANT] ==== -Enabling online access and code execution involves a security risk. Only use these configurations if you trust the source(s). +Enabling online access and code execution introduces security risk (for example, untrusted code execution, outbound data exfiltration). +Only enable these settings when strictly required, and prefer allowlisted sources with restricted egress (for example, NetworkPolicy/EgressFirewall). ====
39-53: Clarify the “Effect” of theopendatahub.io/managedannotation“Allows the user to manage the TrustyAI operator” is vague. The practical effect is about reconcilers not overwriting manual edits to the resource.
| `opendatahub.io/managed` | `true` -| User changes are not allowed and will be automatically reverted. +| The ODH reconciler manages this resource; manual changes may be reverted automatically. @@ | `opendatahub.io/managed` | `false` -| Allows the user to manage the TrustyAI operator. +| Disables ODH reconciliation for this resource, allowing manual changes to persist.
62-76: Clarify that ConfigMap keys act as cluster-level gates; per-job flags are still requiredAvoid the impression that setting the ConfigMap alone enables the features by default for all jobs.
| `lmes-allow-online` | `false` -| Enables internet access for LMEval jobs at the cluster level. +| Permits (gates) internet access for LMEval jobs cluster-wide. +| Effective only when `spec.allowOnline` is also set on the individual `LMEvalJob`. @@ | `lmes-allow-code-execution` | `false` -| Allows execution of code downloaded alongside datasets or tokenizers. +| Permits (gates) remote code execution for artifacts downloaded by LM-Eval (for example, datasets, tokenizers). +| Effective only when `spec.allowCodeExecution` is also set on the `LMEvalJob`.
30-31: Avoid hard-coding the operator namespace in multiple commandsConsider defining a namespace attribute once (for example,
:odh-app-namespace: redhat-ods-applications) and using-n {odh-app-namespace}across commands to reduce duplication and ease maintenance.Also applies to: 58-59, 81-82
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc(1 hunks)modules/setting-up-lmeval.adoc(2 hunks)modules/updating-lmeval-job-configuration-using-the-cli.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-console.adoc(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc
- modules/setting-up-lmeval.adoc
- modules/updating-lmeval-job-configuration-using-the-console.adoc
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: bredamc
PR: opendatahub-io/opendatahub-documentation#860
File: modules/using-llm-as-a-judge-metrics-with-lmeval.adoc:62-65
Timestamp: 2025-07-16T09:05:50.422Z
Learning: In the opendatahub-documentation project, bredamc prefers comprehensive error detection in code reviews. Continue to point out potential errors whether they are related to the code changes in the PR or not, but clearly distinguish between errors introduced in the PR versus errors in existing code.
🔇 Additional comments (3)
modules/updating-lmeval-job-configuration-using-the-cli.adoc (3)
26-37: ✅ Verified correct ConfigMap for LMEval settingsA repo-wide search confirms that:
- The
trustyai-service-operator-configConfigMap is explicitly documented as the source for LM-Eval global settings (see modules/setting-up-lmeval.adoc:38).- Several LMEval modules reference and patch this ConfigMap (e.g. updating-lmeval-job-configuration-using-the-cli.adoc).
- The
opendatahub.io/managedannotation is used throughout to toggle operator management.No changes required.
77-82: Ignore incorrect operator suggestion—TrustyAI operator manages LMEvalJob
LMEvalJob CRDs are reconciled by the TrustyAI Kubernetes operator (see modules/lmeval-evaluation-job.adoc), so restarting thetrustyai-service-operator-controller-managerdeployment is the correct step. No changes required.Likely an incorrect or invalid review comment.
54-61: Double-check LMEval ConfigMap settings
- The keys
lmes-allow-onlineandlmes-allow-code-executionare used consistently across the documentation and match exactly.- These settings are only referenced in docs (e.g. modules/setting-up-lmeval.adoc, modules/updating-lmeval-job-configuration-using-the-cli.adoc).
- There are no code references in this repo to validate the owning component; please manually confirm that the TrustyAI operator’s cluster-level ConfigMap named
trustyai-service-operator-configactually defines these keys for controlling LMEval jobs.
modules/updating-lmeval-job-configuration-using-the-console.adoc
Outdated
Show resolved
Hide resolved
BSynRedhat
force-pushed
the
IENG-301_AllowOfflineText
branch
from
c1ae17a to
f9d521a
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
modules/setting-up-lmeval.adoc (1)
48-48: Typo (pre-existing): fix inline code formattingChange “assign a value for
--device argument” to “assign a value for the--deviceargument”.
🧹 Nitpick comments (4)
modules/setting-up-lmeval.adoc (2)
74-80: Defaults aligned and key name is correct; consider upstream/downstream gating and formatting booleans
- Good: Using
lmes-allow-code-executionmatches the agreed key.- If upstream defaults differ, gate the table rows with ifdef/ifndef as needed; otherwise, leaving both as false is fine for downstream.
- Minor: Keep boolean defaults consistently formatted as code literals.
Suggested optional gating and formatting adjustments:
+ifdef::upstream[] | `lmes-allow-online` -| false +| `true` | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. +endif::[] +ifndef::upstream[] | `lmes-allow-online` -| false +| `false` | Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. +endif::[] - | `lmes-allow-code-execution` - | false +ifdef::upstream[] +| `lmes-allow-code-execution` +| `true` +| Determines whether LMEval jobs can set the `trust remote code` mode to `on`. +endif::[] +ifndef::upstream[] +| `lmes-allow-code-execution` +| `false` | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. +endif::[]If upstream is not meant to be enabled by default, simply apply the code formatting part:
-| false +| `false` ... -| false +| `false`Note: Consistency with prior guidance that the correct key is
lmes-allow-code-executionis maintained (per team learning). Would you like me to push the gated version if upstream truly differs?
86-89: Gate the IMPORTANT note to downstream if upstream defaults differThe note is correct for product docs. If upstream defaults are not disabled, wrap this in
ifndef::upstream[]to avoid confusing upstream readers.+ifndef::upstream[] [IMPORTANT] -- The `allowOnline` and `allowCodeExecution` settings are *disabled* by default at the operator level in {productname-long}. LMEval jobs do not have internet access or permission to run any externally downloaded code unless explicitly enabled. -- +endif::[]modules/updating-lmeval-job-configuration-using-the-cli.adoc (2)
39-53: Remove stray tab in the table titleThere’s a tab character between the filename and the descriptor; it may render oddly. Replace with a single space.
-.trustyai-service-operator-config annotation keys for LMEval job access +.trustyai-service-operator-config annotation keys for LMEval job access
89-92: Fix placeholder spacing to avoid copy/paste confusionRemove the extra space before the closing
>in the namespace placeholder.-oc patch lmevaljobs example-lmeval -n <user-or-target-namespace > \ +oc patch lmevaljobs example-lmeval -n <user-or-target-namespace> \ ... -oc get lmevaljobs example-lmeval -n <user-or-target-namespace > \ +oc get lmevaljobs example-lmeval -n <user-or-target-namespace> \
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc(1 hunks)modules/setting-up-lmeval.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-cli.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-console.adoc(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- modules/updating-lmeval-job-configuration-using-the-console.adoc
- assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-08T15:08:58.799Z
Learnt from: RobGeada
PR: opendatahub-io/opendatahub-documentation#901
File: modules/setting-up-lmeval.adoc:102-104
Timestamp: 2025-08-08T15:08:58.799Z
Learning: In opendatahub-io/opendatahub-documentation, the correct ConfigMap key is `lmes-allow-code-execution` (not `lmes-code-execution`); ensure consistency across all modules and examples.
Applied to files:
modules/setting-up-lmeval.adoc
📚 Learning: 2025-07-16T09:05:50.422Z
Learnt from: bredamc
PR: opendatahub-io/opendatahub-documentation#860
File: modules/using-llm-as-a-judge-metrics-with-lmeval.adoc:62-65
Timestamp: 2025-07-16T09:05:50.422Z
Learning: In the opendatahub-documentation project, bredamc prefers comprehensive error detection in code reviews. Continue to point out potential errors whether they are related to the code changes in the PR or not, but clearly distinguish between errors introduced in the PR versus errors in existing code.
Applied to files:
modules/setting-up-lmeval.adoc
🔇 Additional comments (2)
modules/updating-lmeval-job-configuration-using-the-cli.adoc (2)
58-60: Good: correct ConfigMap keys and string-valued booleansPatching
lmes-allow-onlineandlmes-allow-code-executionwith string booleans is correct for ConfigMap data.
69-76: Defaults table consistent with product defaultsDefaults shown as
falsefor both keys align with the downstream/product stance.
modules/updating-lmeval-job-configuration-using-the-console.adoc
Outdated
Show resolved
Hide resolved
modules/updating-lmeval-job-configuration-using-the-console.adoc
Outdated
Show resolved
Hide resolved
modules/updating-lmeval-job-configuration-using-the-console.adoc
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
modules/setting-up-lmeval.adoc (1)
78-80: Update deprecated ConfigMap key and confirm defaults
Found one remaining use of the old key in
modules/using-llm-as-a-judge-metrics-with-lmeval.adoc:
Line 68:
"lmes-code-execution": "true",
should be
"lmes-allow-code-execution": "true",All other references to
lmes-allow-code-executionare correct:• modules/setting-up-lmeval.adoc (defaults to
false)
• modules/updating-lmeval-job-configuration-using-the-cli.adoc
• modules/updating-lmeval-job-configuration-using-the-console.adocIn setting-up-lmeval.adoc, the default of
falseis appropriate.
If upstream examples should default totrue, wrap the table rows in anifdef::upstream[]block, e.g.:ifdef::upstream[] | `lmes-allow-code-execution` | `true` | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. endif::[] ifndef::upstream[] | `lmes-allow-code-execution` | `false` | Determines whether LMEval jobs can set the `trust remote code` mode to `on`. endif::[]Please update the deprecated key and consider gating defaults if needed.
🧹 Nitpick comments (2)
modules/setting-up-lmeval.adoc (2)
48-48: Tighten code formatting for the CLI flag.Use “the
--deviceargument” (don’t include “argument” inside backticks).-| Detect if there are GPUs available and assign a value for the `--device argument` for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`. +| Detect if there are GPUs available and assign a value for the `--device` argument for LM Evaluation Harness. If GPUs are available, the value is `cuda`. If there are no GPUs available, the value is `cpu`.
74-77: Confirm intended defaults; consider upstream/downstream gating for allow-online.Table sets
lmes-allow-onlinedefault to false unconditionally. If upstream defaults should be enabled (as previously discussed), gate the rows by context; otherwise keep false for both.-| `lmes-allow-online` -| `false` -| Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. +ifdef::upstream[] +| `lmes-allow-online` +| `true` +| Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. +endif::[] +ifndef::upstream[] +| `lmes-allow-online` +| `false` +| Whether LMEval jobs can set the online mode to `on` to access artifacts (models, datasets, tokenizers) from the internet. +endif::[]
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc(1 hunks)modules/setting-up-lmeval.adoc(2 hunks)modules/updating-lmeval-job-configuration-using-the-cli.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-console.adoc(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- modules/updating-lmeval-job-configuration-using-the-cli.adoc
- assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc
- modules/updating-lmeval-job-configuration-using-the-console.adoc
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-08T15:08:58.799Z
Learnt from: RobGeada
PR: opendatahub-io/opendatahub-documentation#901
File: modules/setting-up-lmeval.adoc:102-104
Timestamp: 2025-08-08T15:08:58.799Z
Learning: In opendatahub-io/opendatahub-documentation, the correct ConfigMap key is `lmes-allow-code-execution` (not `lmes-code-execution`); ensure consistency across all modules and examples.
Applied to files:
modules/setting-up-lmeval.adoc
📚 Learning: 2025-07-16T09:05:50.422Z
Learnt from: bredamc
PR: opendatahub-io/opendatahub-documentation#860
File: modules/using-llm-as-a-judge-metrics-with-lmeval.adoc:62-65
Timestamp: 2025-07-16T09:05:50.422Z
Learning: In the opendatahub-documentation project, bredamc prefers comprehensive error detection in code reviews. Continue to point out potential errors whether they are related to the code changes in the PR or not, but clearly distinguish between errors introduced in the PR versus errors in existing code.
Applied to files:
modules/setting-up-lmeval.adoc
BSynRedhat
force-pushed
the
IENG-301_AllowOfflineText
branch
from
f95838e to
62a1f79
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (10)
modules/updating-lmeval-job-configuration-using-the-web-console.adoc (3)
28-31: Standardize AsciiDoc source block syntax and add namespace to YAML exampleUse
[source,yaml](no space) consistently, and includemetadata.namespacein the example to avoid user confusion.- [source, yaml] + [source,yaml] ---- opendatahub.io/managed: "false" ---- - [source, yaml] + [source,yaml] ---- lmes-allow-online: "true" lmes-allow-code-execution: "true" ---- - [source,yaml] + [source,yaml] ---- apiVersion: trustyai.opendatahub.io/v1alpha1 kind: LMEvalJob metadata: name: example-lmeval + namespace: <your-namespace> spec: allowOnline: true allowCodeExecution: true ----Also applies to: 34-38, 55-66
51-54: Clarify which fields must be set per jobSpell out the field names so users know exactly what to set on the CR.
- Each LMEval job must explicitly be set to allow online access and code execution. + Each `LMEvalJob` resource must explicitly set `spec.allowOnline` and/or `spec.allowCodeExecution` to `true`.- . Ensure that the following fields are set to `true` to enable online access and code execution for this job when writing your `LMEvalJob` custom resource: + . When writing your `LMEvalJob` custom resource, set `spec.allowOnline` and/or `spec.allowCodeExecution` to `true` for this job:
41-49: Add note: rollout restart doesn’t affect already running jobsUsers may expect live jobs to pick up changes; this clarifies behavior.
. Click the *Actions* menu and select *Restart rollout*. + [NOTE] -- Each LMEval job must explicitly be set to allow online access and code execution. -- ++ +[NOTE] +-- +The rollout restart updates the operator. Already running `LMEvalJob` pods will not be mutated; submit new jobs or re-run existing ones to pick up the new cluster-level settings. +--modules/updating-lmeval-job-configuration-using-the-cli.adoc (5)
6-6: Normalize title casing to match related moduleUse “job” (lowercase) for consistency with the web console module title.
-= Updating LMEval Job configuration using the CLI += Updating LMEval job configuration using the CLI
21-22: Use the appropriate product reference for the OpenShift CLIThe link targets the OpenShift CLI (oc). Prefer explicit naming or {productname-long} to avoid ambiguity.
-* You have downloaded and installed the {productname-short} command-line interface (CLI). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^]. +* You have downloaded and installed the OpenShift CLI (`oc`). See link:https://docs.redhat.com/en/documentation/openshift_container_platform/{ocp-latest-version}/html/cli_tools/openshift-cli-oc#installing-openshift-cli[Installing the OpenShift CLI^].
46-53: Clarify the effect of opendatahub.io/managed=falseMake the behavior explicit so users understand implications.
-| `opendatahub.io/managed` -| `false` -| Allows the user to manage the TrustyAI operator. +| `opendatahub.io/managed` +| `false` +| Disables ODH auto-management for this ConfigMap so user changes persist. Use with caution.
58-60: Show independent toggling and disabling of settingsDemonstrate setting the keys independently and how to disable later.
oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ --type merge -p '{"data":{"lmes-allow-online":"true","lmes-allow-code-execution":"true"}}' ---- +---- ++ +[source,sh] +---- +# Example: enable only online access +oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ + --type merge -p '{"data":{"lmes-allow-online":"true"}}' + +# Example: disable code execution later +oc patch configmap trustyai-service-operator-config -n redhat-ods-applications \ + --type merge -p '{"data":{"lmes-allow-code-execution":"false"}}' +----
77-82: Optional: validate operator rollout completionHelps users ensure the change has been applied.
oc rollout restart deployment trustyai-service-operator-controller-manager -n redhat-ods-applications ---- ++ +[source,sh] +---- +# Optional: wait for the rollout to complete +oc rollout status deployment trustyai-service-operator-controller-manager -n redhat-ods-applications +----assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc (2)
10-10: Tighten wording and name the “web console” explicitlySmall clarity improvement.
-Follow the steps below to enable online access and remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. +Follow the steps below to enable online access and/or remote code execution for LMEval jobs. Choose to update these settings using either the CLI or the web console.
7-11: Add a top-level security warningSurface the risk at the assembly level so readers don’t miss it.
[role='_abstract'] LMEval jobs do not allow internet access or remote code execution by default. When configuring an `LMEvalJob`, it may require access to external resources, for example task datasets and model tokenizers, usually hosted on link:https://huggingface.co[Hugging Face]. If you trust the source and have reviewed the content of these artifacts, an `LMEvalJob` can be configured to automatically download them. -Follow the steps below to enable online access and remote code execution for LMEval jobs. Choose to update these settings by using either the CLI or in the console. Enable one or both settings according to your needs. +Follow the steps below to enable online access and/or remote code execution for LMEval jobs. Choose to update these settings using either the CLI or the web console. + +[IMPORTANT] +==== +Enabling online access or remote code execution introduces security risk. Only enable these settings when you trust and have reviewed the external sources. +====
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
.DS_Storeis excluded by!**/.DS_Store
📒 Files selected for processing (5)
assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc(1 hunks)assemblies/evaluating-large-language-models.adoc(1 hunks)modules/setting-up-lmeval.adoc(2 hunks)modules/updating-lmeval-job-configuration-using-the-cli.adoc(1 hunks)modules/updating-lmeval-job-configuration-using-the-web-console.adoc(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- assemblies/evaluating-large-language-models.adoc
- modules/setting-up-lmeval.adoc
🔇 Additional comments (4)
modules/updating-lmeval-job-configuration-using-the-web-console.adoc (1)
75-82: Table descriptions look good; defaults align with procedural stepsNo changes needed.
modules/updating-lmeval-job-configuration-using-the-cli.adoc (2)
89-93: Good: plural resource name and validation stepCorrect use of
lmevaljobsand the jsonpath verification is clear.
98-108: YAML example looks correct and aligned with other modulesapiVersion, kind, and spec flags are consistent. Including
metadata.namespaceis helpful.assemblies/enabling-external-resource-access-for-lmeval-jobs.adoc (1)
12-15: Includes are correct and ordering is appropriateNo changes needed.
|
LGTM! |
Update to insert a section between current 6.1 and 6.2 about allowing offline access to online and external code sources.
Summary by CodeRabbit